Performance Study of the MPI and MPI-CH Communication Libraries on the IBM SP

During our research in the simulation of message-passing applications on parallel, high-performance systems, we have discovered that the native MPI implementation on the IBM SP suffers from performance anomalies. Our simulator, MPI-Sim, predicted a smooth performance for an ASCI relevant application, whereas the measured system showed a sudden jump in the runtime of the application. Surprisingly, the MPI-CH implementation on the same machine does not suffer from the same performance degradation. This report identifies the anomaly and summarizes the relative performance results for MPI and MPI-CH for SWEEP3D, a large scientific application.